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ABSTRACT 

Background: Death or urgent readmission after hospital discharge is a common adverse event that can be used to 
compare outcomes of care between institutions. To accurately adjust for risk and to allow for interhospital compari- 
sons of readmission rates, we used administrative data to derive and internally validate an extension of the LACE 
index, a previously validated index for 30-day death or urgent readmission. 

Methods: We randomly selected 500000 medical and surgical patients discharged to the community from any On- 
tario hospital between 1 April 2003 and 31 March 2009. We derived a logistic regression model on 250 000 randomly 
selected patients from this group and modified the final model into an index scoring system, the LACE+ index. We 
internally validated the LACE+ index using data from the remaining 250 000 patients and compared its performance 
with that of the original LACE index. 

Results: Within 30 days of discharge to the community, 33 825 (6.8%) of the patients had died or had been urgently 
readmitted. In addition to the variables included in the LACE index (length of stay in hospital [L], acuity of admis- 
sion [A], comorbidity [C] and emergency department utilization in the 6 months before admission [E]), the LACE+ 
index incorporated patient age and sex, teaching status of the discharge hospital, acute diagnoses and procedures 
performed during the index admission, number of days on alternative level of care during the index admission, and 
number of elective and urgent admissions to hospital in the year before the index admission. The LACE+ index was 
highly discriminative (C statistic 0.771, 95% confidence interval 0.767-0.775), was well calibrated across most of its 
range of scores and had a model performance that exceeded that of the LACE index. 

Interpretation: The LACE+ index can be used to predict the risk of postdischarge death or urgent readmission on the 
basis of administrative data for the Ontario population. Its performance exceeds that of the LACE index, and it allows 
analysts to accurately estimate the risk of important postdischarge outcomes. 
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Death or urgent readmission after hospital 
discharge is a relatively common event that is costly to 
the health care system and has obvious important ef- 
fects on patients' health. Being able to accurately identify 
patients at high risk of these adverse postdischarge out- 
comes could help to elucidate the mechanisms involved 
in early death or readmission. In addition, accurately 
adjusting for the risk of postdischarge death or urgent 
readmission may permit valid interhospital comparisons 
based on this outcome. 

Several risk prediction models for hospital readmis- 
sion have been published. 1 " 4 Three of these models 1-3 
are cumbersome because they require community-level 
or socio-economic information (e.g., community-level 
admission rates, ethnicity, education, postal code or 
marital status) that is difficult or impractical to obtain or 
apply. One model (the LACE index 4 ) includes variables 
for which values can be determined from either primary 
data or information in administrative databases. The 
LACE index uses 4 variables to predict the risk of death 
or urgent readmission within 30 days after hospital dis- 
charge among medical and surgical patients: length of 
hospital stay (L), acuity of admission (A), comorbidity (C) 
and emergency department utilization in the 6 months 
before admission (E). The LACE index was externally 
validated and had good calibration. Its discrimination 
was only fair (C statistic 0.684), though equivalent to that 
of previously published, more complicated models. 1-3 

The LACE index was derived from data for a small 
sample of patients (approximately 2500). 4 The small 
sample size could partly explain the small number of 
variables in the model, as well as the limited amount of 
information about the index hospital admission. In the 
study reported here, we determined whether adminis- 
trative data from a large, population-based sample could 
be used to extend and improve the LACE index. In par- 
ticular, we wanted to develop an index that would allow 
researchers and analysts to use administrative data to 
better predict the risk of postdischarge outcomes. 

Methods 

The study was approved by The Ottawa Hospital Re- 
search Ethics Board. 

Study design. This cohort study involved 500 000 ran- 
domly selected patients who had been discharged to the 
community from a medical or surgical service at any On- 
tario hospital during the defined study period. We de- 
rived the risk index using data from a randomly selected 
sample of 250 000 patients and internally validated the 
index using data from the remaining 250 000 patients. 



Data sources. We used 4 population-based administra- 
tive databases that captured data on all Ontarians. The 
Discharge Abstract Database (DAD) records all non- 
psychiatric admissions to hospital. The Ontario Men- 
tal Health Reporting System (OMHRS) captures all 
inpatient mental health encounters after 2006 (before 
which these admissions were captured in the DAD). The 
National Ambulatory Care Reporting System (NACRS) 
records all emergency department visits. The Registered 
Patient Database (RPDB) records the death date for all 
Ontarians. 

Study cohort. We used the DAD to identify all dischar- 
ges of adults to the community (i.e., excluding discharges 
to rehabilitation and long-term care facilities) from med- 
ical and surgical services in acute care Ontario hospitals 
between 1 April 2003 and 31 March 2009. We chose this 
study period to ensure availability of sufficient data in 
the population-based databases to create the model co- 
variates and determine the study outcome for all patients. 
We excluded data for patients who underwent same-day 
surgeries, because these are similar to outpatient surger- 
ies and involve patients whose characteristics may differ 
from those of inpatients. If a patient was discharged from 
an acute care hospital and was transferred directly to an- 
other acute care hospital (which was the case for about 4% 
of admissions during the study period), we linked the 2 
hospital stays and considered them as a single admission. 
We excluded psychiatric and obstetric admissions be- 
cause they were not included in the original derivation of 
the LACE index. 4 We also excluded admissions where the 
patient was ineligible for health care coverage in Ontario 
during the follow-up period, because postdischarge out- 
comes would not have been captured for these patients. 

After identifying all eligible admissions for the study 
period, we randomly selected one admission per patient 
and then randomly selected 500 000 patients. 

Potential predictors of 30-day death or urgent re- 
admission. We first created the 4 predictors used in the 
LACE index: length of the index hospital stay (L), in days; 
acuity of the index admission (A), categorized as urgent or 
planned; comorbidity of the patient (C), expressed by the 
Charlson score; and emergency department utilization 
(E), expressed as the number of visits to the emergency 
department in the 6 months before the index admis- 
sion. Hospital length of stay and urgency of admission 
were determined from the DAD. The Charlson score was 
calculated from diagnoses for the index admission and 
the International Statistical Classification of Diseases 
and Related Health Problems, 10th revision, Canada 
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(ICD-io-CA) codes cited by Quan and colleagues. 5 The 
number of visits to an emergency department in the 6 
months before the index admission was determined by 
linking to the NACRS. 

We captured important acute diagnoses and pro- 
cedures performed during the index admission with a 
scoring index based on case-mix group (CMG) codes, 
which we previously derived and internally validated 
using 200 ooo admissions randomly selected from the 
same patient population as used for the current study. 6 
The previous study used a smaller sample of admissions 
because of the computationally intensive nature of the 
variable-selection procedure used to derive the CMG 
score. CMGs aggregate acute care inpatients with similar 
clinical and resource-utilization characteristics and are 
assigned by an algorithm developed (and revised yearly) 
by the Canadian Institute for Health Information (CIHI). 
We assigned a score ranging from -6 to 7 to certain CMG 
codes that were independently associated with 30-day 
death or urgent readmission, independent of the LACE 
score. A CMG score above zero was associated with in- 
creased risk, whereas a CMG score below zero was as- 
sociated with decreased risk. We determined the CMG 
code for all patients using the 2008 CMG algorithm ap- 
plied to all hospital admissions. 

Finally, we used data in the DAD to create the follow- 
ing covariates for each hospital admission: patient age 
and sex; number of urgent and elective hospital admis- 
sions in the year before the index admission, expressed 
as separate variables; number of days in an intensive 
care unit (ICU) during the index admission; number of 
days with alternative level of care status during the index 
admission (a status assigned to patients who remain in 
hospital but are no longer receiving active medical care); 
and teaching status of the hospital from which the pa- 
tient was discharged. We further divided nonteaching 
hospitals according to size, where large hospitals were 
those with 100 or more beds, and small hospitals were 
those with fewer than 100 beds. For previous hospital 
admissions, we chose to use a longer look-back period of 
1 year (rather than the 6 months used for previous emer- 
gency department visits in the LACE index) because ad- 
missions are less common than emergency department 
visits and because we separated previous hospital stays 
into urgent and elective categories. We determined the 
number of days spent in an ICU using validated special 
care unit codes for ICU treatment. 7 

For all admissions, there were no missing values for 
any of the potential predictors, since values originated 
as or were derived from required fields in the hospital 
admission abstract. 



Outcome. The primary outcome was death or urgent 
readmission within 30 days of hospital discharge. We 
determined death status by linking to the RPDB. We 
determined urgent readmission status by linking to the 
DAD and OMHRS. We included readmissions regard- 
less of the diagnosis, as long as they were categorized as 
"urgent" (i.e., unplanned) and were not preceded by an 
earlier "non-urgent" (i.e., planned) readmission. 

Derivation of the LACE+ index. We first derived a 
logistic regression model for 30-day death or urgent re- 
admission using data for a randomly selected group of 
250000 patients. We entered all candidate covariates 
into an initial multivariable model and then performed 
variable selection (with a significance level of a = 0.05) 
using methods described by Sauerbrei and Royston. 8 
These methods combined backward selection with a sys- 
tematic process of identifying the optimal first-degree 
fractional polynomial transformation for continuous co- 
variates. We did not force any covariates into the model, 
including the 4 LACE predictors. Once we had identified 
all significant covariates, we created a list of candidate 
interaction terms on the basis of clinical intuition, added 
them jointly to the model and used backward selection to 
remove interactions with ap value > 0.0001. 

We then used methods described by Sullivan and col- 
leagues 9 to modify the final logistic regression model 
into a risk index, which we refer to as the LACE+ index. 
The number of points assigned to each binomial variable 
was equal to the covariate's regression coefficient div- 
ided by a constant (the category in the model with the 
smallest absolute value in regression units) and round- 
ed to the nearest integer. For continuous variables, we 
first categorized the variable using decile cut points. The 
number of points assigned to each category was equal 
to the difference in regression units between its mid- 
point and the midpoint of the reference category divided 
by the constant and rounded to the nearest integer. We 
combined continuous-variable categories to which the 
same number of points had been assigned. The reference 
category for each variable in the model was assigned o 
points. 

Assessment and internal validation of the LACE+ 
index. Using data for the remaining 250000 patients 
(the validation set), we assessed the ability of the LACE+ 
index to predict 30-day death or urgent readmission. We 
first calculated each patient's LACE+ index. We then fit- 
ted a logistic regression model with 30-day death or ur- 
gent readmission as the outcome and the LACE+ index 
as the independent predictor. We assessed the model's 
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discrimination using the C statistic with 95% confi- 
dence intervals (CIs) 10 and assessed overall calibration 
using the Hosmer-Lemeshow (H-L) goodness-of-fit test 
with 8 degrees of freedom. 11 The H-L test compares the 
observed to expected event rates within risk deciles of 
the population, where a nonsignificant test statistic (p > 
0.05) suggests good calibration. We further assessed 
calibration by comparing the expected and observed 
event rates within 10-point strata. The expected risk of 
death or urgent readmission for each patient was calcu- 
lated as the inverse of 1 + e - (intercept + P * LACE+ score) , where (3 
was the coefficient of the LACE+ index in the regression 
model. The expected and observed event rates were con- 
sidered similar if the expected rate was within the exact 
95% CI 12 around the observed rate. 

In the validation set, we compared the performance 
of the original LACE index with that of the LACE+ index 
using the integrated discrimination improvement (IDI) 
and the net reclassification improvement (NRI). 13 The 
IDI is the discrimination slope (the mean predicted risk 
for patients with the event minus that for patients with- 
out the event) of a model with the LACE+ index as the 
independent predictor minus the discrimination slope of 
a model with the LACE index as the independent pre- 
dictor. An IDI above zero indicates better discrimination 
(i.e., a larger separation in mean predicted risk between 
events and nonevents) with the LACE+ index. The NRI 
measured the amount of correct reclassification (i.e., 
predicted risk moving upward for 
events and downward for nonevents) 
when the predicted risk of death or ur- 
gent readmission based on the LACE+ 
index was compared with that based 
on the LACE index. The NRI equals 
the proportion of correct minus in- 
correct reclassifications among events 
(i.e., patients who died or were urgent- 
ly readmitted) plus the proportion of 
correct minus incorrect reclassifica- 
tions among nonevents. An NRI above 
zero indicates better risk prediction 
with the new (i.e., LACE+) model. Be- 
cause no established risk categories 
for early death or urgent readmission 
exist, we calculated a category-less 
NRI, 14 where "upward" reclassifica- 
tion was defined as a higher event risk 
predicted by the LACE+ index and 

"downward" reclassification was de- 
Figure 1 

fined as a lower event risk predicted Outline for creation of study cohort. OHIP = Ontario Health Insurance Plan 
by the LACE+ index. 



Results 

Of approximately 6.5 million hospital admissions in On- 
tario during the study period, nearly 3.3 million were eli- 
gible for the study (Figure 1). These admissions involved 
more than 1.8 million individuals, of whom 500000 
were randomly selected for the study. 

Patients were generally middle-aged and had few 
documented chronic comorbidities (Table 1). Approxi- 
mately one-third had visited the emergency department 
in the previous 6 months; fewer had been admitted to 
hospital urgently (14.0%) or electively (6.2%) in the past 
year. About half of the admissions had been to a large 
nonteaching hospital. In nearly one-third of the admis- 
sions, the patient had an acute diagnosis or procedure as- 
sociated with an increased (CMG score > o) or decreased 
(CMG score < o) risk of 30-day death or urgent readmis- 
sion. Admission to an ICU (11.4% of patients) or transfer 
to an alternative level of care (1.9%) was uncommon dur- 
ing the index admission. After discharge to the commun- 
ity, 6.8% of the patients died or were urgently readmitted 
within 30 days. Only 0.7% of patients died within 30 days 
without a prior urgent readmission. The derivation and 
validation sets were virtually identical (Table 1). 

With the exception of number of days in an ICU, all 
covariates that we considered were significantly associ- 
ated with 30-day death or urgent readmission and were 
included in the final model (Textbox 1). Of the continuous 
covariates, all except number of days on alternative level 



All acute care hospital 
admissions in Ontario between 
1 Apr. 2003 and 31 Mar. 2009 
n = 6516313 






Excluded n = 3239280 

• died in hospital n =261 336 

• age < 1 8 years n = 1 296 820 

• psychiatric or obstetric admission n = 1 021 732 

• discharged to long-term care, rehabilitation or 
other hospital n = 656 011 








Eligible hospital admissions 
n = 3 277 033 




• ineligible for OHIP coverage at discharge or 
during 30-day postdischarge period n = 3381 



i 



Random selection of 1 admission 
per person 
n= 1851 930 



Random selection of final cohort 
n = 500 000 
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of care were modelled with either a logarithmic, square 
root, inverse or squared transformation (Table 2; Figure 
2). Six interaction terms were applied to this model: age 
combined with Charlson score, number of urgent admis- 
sions in the previous year, CMG score and length of stay; 
urgency of admission combined with CMG score; and 
Charlson score combined with number of urgent admis- 
sions in the previous year. Of these, 3 were highly signifi- 
cant (p < 0.0001) and were ultimately retained (Table 2; 
Figure 2). 

We then modified the final logistic regression model 
into a scoring index for 30-day death or urgent readmis- 
sion (Table 3). For patient age, the number of points 
assigned depended on the Charlson score and the num- 
ber of urgent admissions in the previous year. The final 
LACE+ index ranged from -15 to 114. In the validation 
set, the distribution of the LACE+ index was skewed 
slightly to the right (Figure 3). 

The LACE+ index was significantly associated with 
the risk of 30-day death or urgent readmission, and the 
odds ratio for a l-point increase in the LACE+ index was 
1-045 (95% CI 1.044-1.045). The index was highly dis- 
criminative, indicated by a C statistic of 0.771 (95% CI 
0.767-0.775), which significantly exceeded that of the 
LACE index for the same population (C statistic 0.738, 
95% CI 0.734-0.741). The H-L test for the LACE+ index 
was highly significant (H-L statistic 31.93, p < 0.0001), 
which suggests poor calibration; conversely, the H-L sta- 
tistic for the LACE index was not significant (H-L sta- 
tistic 13.38, p = 0.10), which suggests good calibration. 
However, the calibration graph comparing the observed 
and expected event rates within strata of the LACE+ 
index showed that the calibration of the LACE+ index 
was in fact quite good. The observed and expected rates 
were very similar in all but 2 of the highest-risk strata 
(Figure 3). These high-risk strata (representing patients 
with a LACE+ index between 90 and 109) constituted 
only about 1.6% of the validation set (Table 4). Across all 
risk strata, the overall expected rate was identical with 
the observed rate (Table 4). 

The LACE+ index was significantly associated with 
each outcome separately. For death within 30 days, the 
odds ratio for a l-point increase in the LACE+ index was 
1.068 (95% CI 1.066-1.069). The index had excellent 
discrimination for 30-day death (C statistic 0.883, 95% 
CI 0.879-0.888), but the H-L test suggested that it was 
poorly calibrated (H-L statistic 114.24, p < 0.0001). For 
urgent readmission within 30 days, the odds ratio for 
a l-point increase in the LACE+ index was 1.040 (95% 
CI 1.040-1.041). The index had good discrimination 
for 30-day urgent readmission (C statistic 0.753, 95% 



CI 0.749-0.757), but again, the H-L test suggested poor 
calibration (H-L statistic 58.93, p < 0.0001). 

The IDI and NRI both indicated that the LACE+ index 
was significantly better at predicting 30-day death or ur- 
gent readmission than the original LACE index. The IDI 
was 0.020 (95% CI 0.019-0.021) on the absolute scale 
and 16.7% (15. 3%-i8.2%) on the relative scale, both of 
which suggested that overall discrimination was better 
with the LACE+ index than with the LACE index. The 
NRI was 0.336 (95% CI 0.321-0.352), which indicated 
that a significantly higher proportion of patients were 
correctly reclassified with the LACE+ index. 

Because the CMG score cannot be calculated for hos- 
pital admissions outside Canada, we conducted a sec- 
ondary analysis to determine the performance of the 
LACE+ index without the CMG score for potential use 
of this index in other countries. When the CMG score 
was assigned a value of o points for all patients, the dis- 
criminative ability of the LACE+ index for predicting 30- 
day death or urgent readmission was only slightly worse 
(Table 5). When each outcome was analyzed separately, 
removing the CMG score was more detrimental for pre- 
dicting 30-day death than for predicting 30-day urgent 
readmission (Table 5). 

Textbox 1 

Components in the LACE and LACE+ indices 
LACE index 

Covariates 

Length of stay 
Acuity of admission 

Comorbidity of the patient (Charlson score) 
Emergency department utilization in previous 6 months 

LACE+ index 

Covariates 

Length of stay 
Acuity of admission 

Comorbidity of the patient (Charlson score) 
Emergency department utilization in previous 6 months 
Age 
Sex 

Teaching status of discharge institution 
Number of urgent admissions in previous year 
Number of elective admissions in previous year 
Case-mix group score 

Number of days on alternative level of care status 

Interaction terms 

Age x Charlson score 

Age x number of urgent admissions in previous year 
Charlson score x number of urgent admissions in previous year 
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Interpretation 

In this study, we derived and internally validated a new 
index, the LACE+ index, for 30-day death or urgent re- 
admission that can be calculated from administrative 
data. This index had excellent discrimination and was 
well calibrated across most of its range. 

The LACE+ index contained all of the predictors in- 
cluded in the LACE index, but it also captured infor- 
mation on patient age and sex, acute conditions and 
procedures performed during the index admission, 
whether the patient changed from active care to an al- 
ternative level of care during the admission, the teaching 
status of the discharge hospital and the number of ur- 
gent and elective admissions in the year before the index 
admission. Previous admissions were not found to be a 
significant predictor of 30-day death or urgent readmis- 
sion in the derivation of the LACE index. 4 The fact that 
previous admissions were a significant independent pre- 
dictor in this study was likely due to the longer look-back 
Table 1 

Description of study cohort 



period and the larger sample size. When these new co- 
variates were combined with the predictors from the 
LACE index, the ability to predict risk of 30-day death or 
urgent readmission improved significantly. 

Given the large size of the validation set, we were 
not surprised that the H-L statistic (for calibration) was 
highly significant for the LACE+ index. With very large 
sample sizes, even a slight departure from a perfect fit 
can result in significance with this test. 15 Therefore, as 
suggested by Kramer and Zimmerman, 15 we supple- 
mented the results of the H-L test by constructing a cali- 
bration graph comparing the observed and predicted 
event rates within 10-point strata for the LACE+ index. 
The calibration graph revealed close agreement between 
the observed and predicted event rates in all but the 
highest-risk strata, which accounted for only a very small 
proportion of patients. Although the 95% CI around the 
observed event rate excluded the expected rate for 5 of 
the strata (Figure 3), 2 of these strata contained a large 





Group; no. (%) of patients* 




Characteristic 


Overall Derivation 
(n = 500 000) (n = 250 000) 


Validation 
(n = 250 000) 



Predictors 



Mean age (SD) 


57.9 


(18.4) 


57.9 


(18.4) 


57.9 


(18.5) 


Sex, male 


239 


590 


(47.9) 


120 031 


(48.0) 


119559 


(47.8) 


Charlson score > 0 


124 


447 


(24.9) 


62 368 


(24.9) 


62 079 


(24.8) 


Median length of stay (days) (IQR) 




3 


(2-6) 


3 


(2-6) 


3 


(2-6) 


Urgent admission 


321 


977 


(64.4) 


160 990 


(64.4) 


1 60 987 


(64.4) 


>1 ED visit in previous 6 months 


188 


315 


(37.7) 


94255 


(37.7) 


94 060 


(37.6) 


>1 urgent admission in previous year 


69 


975 


(14.0) 


35 000 


(14.0) 


34 975 


(14.0) 


>1 elective admission in previous year 


30 


750 


(6.2) 


15 200 


(6.1) 


15 550 


(6.2) 


CMG score 
















>0 


78 


053 


(15.6) 


39142 


(15.7) 


38 911 


(15.6) 


<0 


69 


151 


(13.8) 


34554 


(13.8) 


34 597 


(13.8) 


Admitted to an ICU during index admission 


56 


840 


(11.4) 


28 351 


(11.3) 


28 489 


(11.4) 


Median length of stay in ICU (days)t (IQR) 




3 


(2-4) 


3 


(2-4) 


3 


(2-4) 


Switched to ALC status during index admission 


9 


458 


(1.9) 


4661 


(1.9) 


4 797 


(1.9) 


Median number of days on ALC (days)^ 




7 


(3-15) 


7 


(3-15) 


7 


(3-15) 


Discharge institution 
















Teaching hospital 


159 


580 


(31.9) 


79 850 


(31.9) 


79 730 


(31.9) 


Large nonteaching hospital (>100 beds) 


269 


247 


(53.8) 


134469 


(53.8) 


1 34 778 


(53.9) 


Small nonteaching hospital (<1 00 beds) 


71 


173 


(14.2) 


35 681 


(14.3) 


35 492 


(14.2) 


Outcomes 
















Death or urgent readmission within 30 days 


33 


825 


(6.8) 


16 820 


(6.7) 


17 005 


(6.8) 


Urgent readmission within 30 days 


30 


234 


(6.0) 


15 044 


(6.0) 


15190 


(6.1) 


Death within 30 days§ 


3 


591 


(0.7) 


1 776 


(0.7) 


1 815 


(0.7) 



ALC = alternative level of care, CMG = case-mix group, ED = emergency department, ICU = intensive care unit, 
IQR = interquartile range, SD = standard deviation 

* Unless otherwise indicated. 

t Among patients admitted to an ICU during index admission 
£ Among patients switched to ALC during index admission 
§ Not preceded by an urgent readmission 
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number of patients and thus had an extremely narrow CI 
around the observed rate (Table 4). 

We used several methods to compare the performance 
of the LACE+ index and the LACE index. The C statis- 
tic, IDI and NRI all indicated that the performance of 
the LACE+ index was superior. The H-L statistic sug- 
gested that calibration of the LACE index was better, but 
this was likely because the LACE+ index had better dis- 
crimination (calibration often worsens when discrimina- 
tion improves). The C statistic of 0.738 for the LACE 
index in this study was notably higher than the C statistic 
of 0.684 reported from external validation of the LACE 
index in the original study. 4 This discrepancy was most 
likely due to measurement errors in the original study, 
in which transfer admissions were not linked as a single 
admission (instead, transfer admissions in the original 
study were based solely on transfer codes in the DAD). As 
a result, the rate of urgent readmissions was higher in the 
original study than in the current study (7.3% v. 6.0%). 

A major advantage of the LACE+ index is that all of its 
components can be readily determined from data avail- 
able in Canadian administrative databases. The index 



therefore has great utility for researchers using such 
databases, because it could be used as a risk adjustment 
methodology in analyses involving large patient popu- 
lations where early death or readmission to hospital is 
an outcome. Furthermore, the LACE+ index does not 
require the use of community-level or socio-economic 
information, which is often difficult to obtain and may 
be unavailable for certain patients. 

We believe that the LACE+ index could be used in 
several ways. First, it could be used to adjust for import- 
ant covariates when comparing postdischarge outcomes 
between institutions. More specifically, the LACE+ 
index could use routinely collected data to generate the 
expected number of outcomes from particular institu- 
tions, which could be compared with observed outcomes 
to gauge performance. Second, the LACE+ index could 
be used to adjust for trends in important predictors of 
postdischarge outcomes when outcomes from the same 
institution are examined over time. This type of adjust- 
ment would be especially helpful in studies of the effect 
of particular interventions or other events on postdis- 
charge outcomes. 



Table 2 

Final risk prediction model 



Covariate* 


Parameter estimate (SE) 


Adjusted odds 
ratio (95% Cl)t 


p value 


Male 


0.10422 (0.01690) 


1.11 (1.07-1.15) 


<.0001 


Urgent admission 


0.60273 (0.02333) 


1.83(1.75-1.91) 


<.0001 


Discharge institution^ 






0.0067§ 


Teaching vs small non-teaching hospital 


-0.01328 (0.02600) 


0.99 (0.94-1.04) 




Large vs small non-teaching hospital 


-0.06150 (0.02384) 


0.94 (0.90-0.99) 




Age 2 


0.00032 (0.00002) 


NA 


<.0001 


Log (length of stay) 


0.28249 (0.01147) 


NA 


<.0001 


Charlson score 05 


1.32586 (0.07319) 


NA 


<.0001 


Log (number of ED visits in previous 6 months) 


0.37177 (0.01622) 


NA 


<.0001 


Number of urgent admissions in previous year 05 


1.81390 (0.09126) 


NA 


<.0001 


Number of elective admissions in previous year 1 


-0.50616(0.05751) 


NA 


<.0001 


CMG score 2 


0.01393 (0.00031) 


NA 


<.0001 


Number of days on ALC status 


-0.01 033 (0.00209) 


0.99 (0.99-0.99) 


<.0001 


Age 2 x Charlson score 05 


-0.00005 (0.00001) 


NA 


<.0001 


Age 2 x Number of urgent admissions in previous year 05 


-0.00011 (0.00001) 


NA 


<.0001 


Charlson score 05 x Number of urgent admissions in previous year 05 


-0.31468 (0.04856) 


NA 


<.0001 



ED = emergency department, CMG = case mix group, ALC = alternative level of care, CI = confidence interval, NA = not applicable 

* In the final risk prediction model, both male and urgent admission were expressed as binary variables. Discharge institution was a categorical variable 
with three levels expressed using two binary variables {with small non-teaching hospitals designated as the reference level). All other covariates were 
expressed as continuous variables using the transformations specified above. Before applying the transformations, a value of 1 was added to the 
original value of length of stay, Charlson score, number of previous ED visits, number of previous urgent admissions, and number of previous elective 
admissions to ensure each covariate's domain was >0. For the CMG score, a value of 7 was added to the original covariate value before applying the 
transformation. 

tThe adjusted odds ratio is shown for covariates that are not transformed and are not involved in an interaction. Since the adjusted odds ratio for 
transformed covariates and covariates involved in an interaction is not directly interpretable, their effects are presented graphically in Figure 2. 

± Small nonteaching hospital = nonteaching hospital with < 100 beds, large nonteaching hospital = nonteaching hospital with > 100 beds 

§p value for the overall {type 3) effect of discharge institution 
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Table 3 

LACE+ scoring system to predict risk of 30-day death 
or urgent readmission 

Predictor Points 

Male sex 3 

Urgent admission 15 

Discharge institution 
Teaching hospital or small nonteaching hospital* 0 
Large nonteaching hospitalt -1 

Length of stay (days) 



<1 


0 


1 


2 


2 


3 


3 


4 


4 


5 


5-6 


6 


7-10 


7 


> 10 


9 


CMG score 




<-2 


-13 


-2 to-1 


-6 


0 


0 


1-2 


8 


>2 


24 



No. of days on ALC status 

0 0 

>0 -1 

No. of ED visits in previous 6 months 

0 0 

1 3 
>1 6 

No. of elective admissions in previous year 

0 0 

>0 6 



Points for age by Charlson score and number of urgent admissions in previous year 



Previous urgent admissions = 0 Previous urgent admissions > 0 



Age (years) 


Charlson 0 


Charlson 1 


Charlson > 1 


Charlson 0 


Charlson 1 


Charlson > 1 


<32 


0 


10 


30 


25 


33 


48 


32-40 


2 


12 


31 


26 


34 


48 


41-46 


5 


15 


34 


27 


35 


49 


47-52 


7 


16 


34 


28 


35 


48 


53-58 


9 


17 


35 


29 


35 


48 


59-64 


12 


20 


38 


30 


36 


49 


65-69 


15 


23 


40 


32 


38 


50 


70-75 


18 


26 


42 


33 


39 


50 


76-80 


20 


27 


42 


35 


40 


50 


>80 


27 


33 


47 


38 


42 


51 



ALC = alternative level of care, ED = emergency department 

* Small nonteaching hospital = nonteaching hospital with < 100 beds 

t Large nonteaching hospital = nonteaching hospital with > 1 00 beds 
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0 1 2 

Number of elective hospital admissions in previous year 



3-2-1 0 1 
CMG score 




0 12 3 4 

Number of urgent hospital admissions in previous year 

Age in years — 44 (25th percentile) — 59 (50th percentile) 
- - 73 (75th percentile) 

0.10 



i 

0 12 3 4 

Number of urgent hospital admissions in previous year 

Charlson score — 0 4 3 -- 12 




2 3 
Charlson score 



Age in years — 44 (25th percentile) 59 (50th percentile) 

- - 73 (75th percentile) 



Figure 2 

Association of transformed continuous covariates and interaction terms in the final logistic regression model (horizontal 
axis) with predicted risk of 30-day death or urgent readmission (vertical axis). The expected risk of death or urgent readmission 
was calculated as the inverse of 1 + e -< mterce P t+ l s, * x ' + ... + P K *x K ) ( wnere p was tne regression coefficient of each covariate (X) in the final 
model, and K was the total number of covariates in the final model. In each graph, the covariate of interest was allowed to vary from 
the 1st to the 99th percentile of observed values, while the value of all other covariates was set to the median or reference value 
(unless otherwise noted). CMG = case-mix group, ED = emergency department. 



Open Medicine 2012;6(3)e98 



Research 



van Walraven et al. 




Figure 3 

Distribution and calibration of the LACE+ 
index by 10-point strata. The bars present the 
number of people in the validation cohort (left 
vertical axis) with each CMG score (horizontal 
axis). The red line presents the expected risk of 
30-day death or urgent readmission (right verti- 
cal axis). The black line presents the observed 
event rate (right vertical axis) with 95% confi- 
dence intervals calculated using exact methods. 7 
The star symbols indicate CMG strata where 
the expected rate excludes the 95% confidence 
interval of the observed rate. 



-10 -10to-l 0-9 10-19 20-29 30-39 40-49 50-59 60-69 70-79 80-89 90-99 100-109 >109 



LACE+ index 



Table 4 

Expected and observed probabi 
in the validation population, by 



lity of 30-day death or urgent readmission 
10-point strata of the LACE+ index 

Probability of death or 
urgent readmission within 30 days, % 



LACE+ index 


No. of patients 


Expected* 


Observed (95% Cl)t 


<-10 


136 


0.60 


0.74 


(0.02-4.03 


-10to-1 


3410 


0.85 


1.11 


(0.79-1 .53) 


0-9 


24 469 


1.25 


1.20 


(1.06-1.34) 


10-19 


36 089 


1.87 


1.59 


(1.47-1.73) 


20-29 


48 666 


2.85 


2.82 


(2.67-2.97) 


30-39 


39 315 


4.30 


4.37 


(4.17-4.58) 


40-49 


31 827 


6.52 


6.48 


(6.21-6.75) 


50-59 


25 001 


9.67 


10.24 


(9.87-10.62) 


60-69 


19 247 


14.21 


14.49 


(14.00-15.00) 


70-79 


11 935 


20.22 


20.03 


(1 9.32-20.76) 


80-89 


5 907 


28.06 


29.63 


(28.46-30.81) 


90-99 


2764 


37.45 


34.33 


(32.56-36.14) 


100-109 


1 200 


47.14 


40.75 


(37.95-43.59) 


>109 


34 


56.22 


55.88 


(37.89-72.81) 


Overall 


250 000 


6.80 


6.80 


(6.70-6.90) 



CI = confidence interval 

*The expected probability was calculated as the predicted risk of 30-day death or urgent readmission 
from the LACE+ score, summed across all patients within each stratum and divided by the number 
of patients in the stratum. 

tThe observed probability was calculated as the number of patients within each stratum who had 
the outcome divided by the number of patients in the stratum. The 95% CIs around the observed 
probability were calculated using exact methods 



This study had several important 
strengths and limitations. First, be- 
cause the LACE+ index was derived 
and validated with a large, randomly 
selected, population-based sample 
from Ontario, we believe that it can 
be confidently applied to all medical 
and surgical patients admitted to 
hospitals in Ontario. However, be- 
fore this index is applied elsewhere 
in Canada, it should be validated in 
other provinces. Second, because 
we had access to population-based 
data sets that included information 
on all deaths, hospital admissions, 
and emergency department visits in 
Ontario, we were able to accurately 
measure death and urgent readmis- 
sion status, as well as previous health 
care utilization characteristics. Re- 
searchers with access to informa- 
tion from only a single health care 
institution may not be able to meas- 
ure the index predictors or postdis- 
charge outcomes as accurately and 
may find that the LACE+ index does 
not perform as well for their patients. 
Third, because CMG codes (deter- 
mined by CIHI) are available only 
for Canadian hospital admissions, 
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the CMG score cannot be calculated 
for administrative data outside of 
Canada. However, given that the dis- 
crimination of the LACE+ index for 
predicting 30-day urgent readmis- 
sion was only slightly worse when 
points from the CMG score were re- 
moved, researchers from other coun- 
tries could apply the LACE+ index 
without the CMG score. Alterna- 
tively, researchers could replace the 
CMG score with data for their own jurisdiction's similar 
index for acute diagnoses and procedures. For example, 
researchers from the United States could substitute a 
scoring index based on the widely used diagnosis-re- 
lated groups developed by the US Centers for Medicare 
and Medicaid Services. Before the LACE+ index could be 
applied in other countries, however, it would have to be 
validated in those countries' respective patient popula- 
tions. As well, because CIHI determines the CMG code 
from retrospectively abstracted hospital data, inclusion 
of the CMG score means that the LACE+ index cannot be 
used with primarily collected data. 

In this study, we derived and internally validated an 
accurate risk index for 30-day death or urgent readmis- 
sion that can be readily determined from administrative 
data. This index could be used as a risk-adjustment tool 
to allow for better inter-hospital comparisons of post- 
discharge outcomes or for patient-level adjustment of 
confounders in studies evaluating the effectiveness of 
postdischarge interventions. 
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Table 5 

Performance of the LACE+ index with and without the CMG score 



Discrimination (C statistic) 



Outcome 

30-day death or urgent readmission 
30-day death 

30-day urgent readmission 
CMG = case-mix group 

* All patients were assigned 0 points for the CMG score. 



LACE+ with CMG score 



0.771 
0.883 
0.753 



LACE+ without CMG score* 

0.759 
0.860 
0.743 
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